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Concurrently Filed Applications 
The following United States patent applications, which were concurrently filed with this 
one on October 28, 1999, are fully incorporated herein by reference: Method and System for 
Navigating a Catheter Probe in the Presence of Field-influencing Objects, by Michael Martinelli, 
Paul Kessman and Brad Jascob; Patient-shielding and Coil System, by Michael Martinelli, Paul 
Kessman and Brad Jascob; Coil Structures and Methods for Generating Magnetic Fields, by Brad 
Jascob, Paul Kessman and Michael Martinelli; Registration of Human Anatomy Integrated for 
Electromagnetic Localization, by Mark W. Hunter and Paul Kessman; System for Translation of 
Electromagnetic and Optical Localization Systems, by Mark W. Hunter and Paul Kessman; 
Surgical Communication and Power System, by Mark W. Hunter, Paul Kessman and Brad 
Jascob; and Surgical Sensor, by Mark W. Hunter, Sheri McCoid and Paul Kessman. 

Field of the Invention 

The present invention is directed generally to image guided surgery, and more 
particularly, to systems and methods for surgical navigation using one or more real-time 
ultrasound images overlaid onto pre-acquired images from other image modalities. 
Description of the Related Art 



Physicians have used pre-acquired ultrasound images to plan surgery for many years. 
Traditionally, ultrasound machines provide two-dimensional images of the relevant human 
anatomy. Physicians use such images to diagnose, among other things, fetal deformities. 
However, until recently, physicians have not used such ultrasound images that have been either 
pre-acquired or acquired in real time during surgery for surgical navigation purposes. 
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Some recent systems permit the use of ultrasound images in conjunction with a 
specialized software running on a computer to plan and execute a surgery. For example, among 
other systems, the Ultraguide system permits a physician to represent an icon representation of a 
surgical instrument on an ultrasound image. This system also plots a trajectory of a surgical 
instrument's probe on a two-dimensional ultrasound image. 

Similarly, the Life Imaging system also provides some help to a physician by converting 
two-dimensional ultrasound images into a three-dimensional cube. Subsequently, the physician 
may view an iconic representation of a surgical instrument on the cube. 

However, none of these systems permit a physician to overlay images from other image 
modalities over ultrasound images along with a display of a surgical instrument on the overlaid 
images. 

SUMMARY OF THE INVENTION 



Objects and advantages of the invention will be set forth in part in the description which 

| follows, and in part will be obvious from the description, or may be learned by practice of the 

i 

1 invention. The objects and advantages of the invention will be realized and attained by means of 

i 

i 

the elements and combinations particularly pointed out in the appended claims. 

One aspect of the present invention is directed to a surgical navigation system comprising 
several devices. In particular, the surgical navigation system includes an ultrasound machine, a 
computer coupled to the ultrasound machine, such that image data corresponding to the images 
acquired by the ultrasound machine can be transferred to the computer. In addition, the surgical 
navigation system includes a memory coupled to the computer which has computer instructions. 
The computer instructions when executed by the computer cause the computer to generate an 



icon representing the surgical instrument with a tip and the surgical instrument's trajectory and to 
overlay the icon on a real-time ultrasound image, such that when the surgical instrument crosses 
the ultrasound image plane the format of the surgical instrument's trajectory is changed to 
represent the surgical instrument's crossing of the ultrasound image's plane. Furthermore, the 
surgical navigation system includes a localizer coupled to the ultrasound machine, which permits 
the system to localize the ultrasound probe, a part of the ultrasound machine. Finally, the 
surgical navigation system includes a display coupled to the computer for displaying the 
generated icon superimposed on the real-time ultrasound image acquired by the ultrasound 
machine. 

The surgical navigation system further includes a display which can display a side view 
of the ultrasound image with a representation of the surgical instrument's trajectory displaying 
the angle at which the surgical instrument's trajectory intersects with the ultrasound image. In 

i 

1 addition, the claimed system can calculate the angle at which the surgical instrument's trajectory 

| 

; intersects with the ultrasound image. Moreover, the claimed system can also represent the angle 

l 

; at which the surgical instrument's trajectory intersects with the ultrasound image using periodic 
| markers. 

Another surgical navigation system consistent with the present invention includes, an 
ultrasound machine and a video imaging device, such as a laparoscope where the ultrasound 
machine and the video-imaging device are coupled to the computer in a way that image data 
from both of these devices can be transferred to the computer. Alternatively, the video imaging 
device may also be an X-ray machine. In addition, the surgical navigation system has localizers 
attached to both the ultrasound machine and to the video imaging device. Furthermore, the 
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: system includes, a memory coupled to the computer, where the memory includes computer 
i instructions. The computer instructions when executed by the computer cause the computer to 

overlay the video images acquired by the video imaging device onto the ultrasound image 
j acquired by the ultrasound device such that the two images correspond to a common coordinate 
j system. Finally, the system includes a display that can display the overlaid images. 
| In addition to the above mentioned systems, the concepts of the present invention may be 

practiced as a number of related methods. 

One method consistent with the present invention is surgical navigation using images 

i 

i 

! from other image modalities overlaid over ultrasound images. The method comprises calibrating 
the system, a step which need be performed initially only; registering image data-sets from other 
' image modalities to patient anatomy, a physician scanning an area of interest using the 
; ultrasound probe; the computer overlaying images from other image modalities onto the 

! ultrasound images; the physician moving the ultrasound probe or other surgical instruments; and 

| 

; the system calculating a new location of the surgical instruments. 

•j Another method consistent with the present invention is a method for calculating and 

j displaying an ultrasound probe's angle relative to an ultrasound image plane. The method 

i 

comprises drawing a circle with its center at the point where the ultrasound probe's trajectory 
crosses the ultrasound image plane. The circle's radius represents the angle of the ultrasound 
probe relative to the ultrasound image plane. The angle of the ultrasound probe relative to the 
ultrasound image plane may also be displayed using periodic markers consistent with the present 
invention. 
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Still another method consistent with the present invention is a method to overlay image 
segmentations onto ultrasound images for surgical navigation. The method comprises extracting 
a two dimensional image from the three-dimensional image data-set; overlaying the extracted 
segmentation onto an ultrasound image corresponding to the same human anatomy; displaying 
the overlaid image along with an iconic representation of the surgical instruments; the physician 
moving the ultrasound probe or other surgical instruments; and the system calculating a new 
location of the surgical instruments. 

An additional method consistent with the present invention is surgical navigation using 
three-dimensional image data-sets. The method comprises a physician acquiring a three- 
dimensional ultrasound image data-set; a computer reconstructing the image data-set into an 
orthogonal data-set; displaying the three-dimensional image on a display along with an iconic 
representation of the surgical instruments; the physician moving the ultrasound probe or other 
surgical instruments; and the system calculating a new location of the surgical instruments. 

Yet another method consistent with the present invention is a method for detecting organ- 
matter shift from the time when CT or MR image data-sets are created to the time when the 
patient is operated upon. The method comprises correlating a real-time ultrasound image and a 
pre-acquired three-dimensional image to obtain a correlated two-dimensional image; selecting a 
first set of points on the real-time ultrasound image; selecting a corresponding second set of 
points on the correlated two-dimensional image; and displaying a vector representing the 
distance and the direction of the organ-matter shift. 
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It is to be understood that both the foregoing general description and the following 
detailed description are exemplary and explanatory only and are not restrictive of the invention, 
as claimed. 

The accompanying drawings, which are incorporated in and constitute a part of this 
specification, illustrate one (several) embodiment(s) of the invention and together with the 
description, serve to explain the principles of the invention. 

BRIEF DESCRIPTION OF THE DRAWINGS 

The accompanying drawings, which are incorporated in and constitute a part of this 
specification, illustrate systems and methods consistent with the invention and, together with the 
description, serve to explain the advantages and principles of the invention. 

FIG. 1 is a diagram of an exemplary system used to overlay ultrasound images onto pre- 
acquired images from other image modalities; 

FIG. 2 is a flow chart illustrating methods consistent with the present invention for 
navigation using overlaid images; 

FIG. 3 is a diagram illustrating a method consistent with the present invention for 
measuring the angle of a surgical instrument's trajectory relative to an ultrasound image plane; 

FIG. 4 is a diagram showing a display of a side view of an ultrasound image with an 
iconic representation of a surgical instrument and its trajectory indicating the angle of the 
surgical instrument relative to the ultrasound probe; 

FIG. 5 is a pictorial image showing an MR data-set image and an ultrasound image side 
by side on a display consistent with the present invention; 
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FIG. 6 is a pictorial image representing a closer view of the MR image and the ultrasound 

image; 

FIG. 7 is a diagram illustrating a method consistent with the present invention for 
displaying the angle of an ultrasound probe relative to an ultrasound image plane; 

FIG. 8 is a diagram illustrating the use of orthogonal ultrasound images in surgical 
navigation consistent with the present invention; 

FIG. 9 is a diagram illustrating a method consistent with the present invention for 
detecting shift in organ matter; 

FIG. 10 is a diagram of an exemplary system used to overlay ultrasound images onto 
video images acquired using a laparoscope or an X-ray machine; 

FIG. 1 1 is a flow chart illustrating methods consistent with the present invention for 
navigation using overlaid ultrasound and video images; 

FIGS. 12 and 13 are pictorial images representing two video laparoscope images of a 
j gallbladder with an ultrasound image overlaid in correct perspective, 
j DETAILED DESCRIPTION 

Reference will now be made in detail to an implementation consistent with the present 
invention as illustrated in the accompanying drawings. Wherever possible, the same reference 
numbers will be used throughout the drawings and the following description to refer to the same 
or like parts. 

j Ultrasound Navigation System Overview 

I Methods and systems consistent with the present invention permit physicians to navigate 

j 

i surgical probes more accurately without relying upon memorized models of the particular 

; 



anatomy they may be operating upon. This is because the present system permits a physician to 
overlay ultrasound images over images from other image modalities giving the physician a three- 
dimensional image of the particular anatomy overlaid onto the ultrasound image relevant to that 
anatomy. Overlaying of images along with correlation of points in images from different image 
modalities can also help the physician in detecting organ-matter shift. 

Fig. 1 is a diagram of an exemplary ultrasound navigation system 100 used to acquire 
ultrasound images. Ultrasound machine 105 is a standard ultrasound machine capable of 
acquiring ultrasound images and displaying them on the built-in monitor. The ultrasound 
machine has a probe 115 attached to it via an arm 1 10, such that the ultrasound probe 1 15 can be 
manipulated by a physician to point it at the area of interest to the physician. For example, the 
physician can point the ultrasound probe 1 15, on the head of a patient to acquire ultrasound 
: images corresponding to the head of that patient. The ultrasound probe 1 15 has a localizer 1 16, 

!l 

;! also referred to as a tracking marker, attached at its distal end. The localizer 1 16 can include 

ij 

!: optical, electromagnetic, acoustic device, or other suitable devices known in the prior art. 

ii 

i! Localization is a standard technique used in image guided surgery to locate the orientation and 

;i 

ii 

position of a surgical instrument or an ultrasound probe relative to a patient's position known in 
the art. For example, each of the following references discuss systems and methods that permit 
localization, each of which is incorporated by reference: PCT Publication WO 96/1 1624 to 
Bucholz et al, published April 25, 1996; U.S. Patent No. 5,384,454 to Bucholz; U.S. Patent No. 
5,851,183 to Bucholz; and U.S. Patent No. 5,871,445 to Bucholz. 

Referring further to Fig. 1, the ultrasound machine 105 is connected to a computer 120 
such that the acquired ultrasound image data can be transferred to the computer 120. The 
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computer 120 is connected to a display 121, such that images stored on a storage media can be 
digitally manipulated, saved, printed, or displayed on the display 121. Three-dimensional 
images, such as pre-acquired images obtained from computed tomography (CT) or Magnetic 
Resonance Imaging (MR) image data-sets for a particular patient, stored on a storage media, such 
as an external tape-drive, not shown, attached to the computer 120 may also be manipulated by 
computer 120 and displayed by the display 121 . In addition, the images displayed on the display 

| 121 may also be displayed through a head-mounted display worn by the physician. 

Additionally, although Fig. 1 shows only one computer 120, one may have multiple 
computers implemented as a single computer to perform the functions performed by the 
computer 120. Moreover, one may not need external storage media for storing the images since 
those images could be stored on a remote server connected to the computer 120 through a local 
area network (LAN). In addition, even though Fig. 1 shows only one display 121 coupled to the 

j computer 120, one may have multiple displays, including LCD displays, connected to the 

i 

i 

j computer 120. 

i 
♦ 

Moreover, even though Fig. 1 only illustrates the use of a patient specific CT/MR data- 

; set, the disclosed system may also have three-dimensional atlas data-sets stored on the computer 

i 

120. For example, one may have a three-dimensional atlas data-set for the human brain or the 
kidney stored on a remote server accessible to the computer 120 through a LAN, or on the 
storage media attached to the computer 120. Atlas data is non-patient specific three-dimensional 
data describing a "generic" patient. Atlas data may be acquired using CT, MR or other imaging 
modalities from a particular patient; and may even comprise images from several modalities 
which are spatially registered (e.g., CT and MR together in a common coordinate system). Atlas 
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data may also have annotations describing anatomy, physiology, pathology, or "optimal" 
planning information. 

In addition, in general, the term "pre-acquired," as used herein, is not intended to imply 
any required minimum duration between the acquisition of image data-sets and displaying the 
corresponding images. Accordingly, a physician may "pre-acquire" image data-sets during 
surgery while operating upon a patient using the disclosed system. 

To operate upon a patient using the disclosed ultrasound surgical navigation system, the 
patient is positioned between the ultrasound probe 1 15 and the tracking sensor 1 17. 

A surgical instrument 140 is also embedded with tracking markers, such as for example, 
emitters and/or reflectors. These markers permit determination of the three dimensional position 
of an object relative to a patient. The determination of the three dimensional position of an 
object relative to a patient is known in the art. As mentioned earlier, each of the Bucholz 

i 

i 

j references discuss systems and methods for determination of the three dimensional position of an 
; object relative to a patient. Using known systems in the prior art, the disclosed system can locate 
j the position and orientation of a surgical instrument or a probe being used to operate upon the 
patient. 

Ultrasound Navigation System Setup and Operation 

Fig. 2 is a flow chart illustrating the steps for methods consistent with the present 
invention for ultrasound navigational guidance using the system of Fig. 1. As shown, there are 
six main steps involved in using the ultrasound navigational guidance system. A physician needs 
to initially perform the first step, Calibrate 201. After calibrating, discussed hereinafter, in the 
second step 203, the pre-acquired data-sets belonging to other image modalities may need to be 
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registered in order to obtain a correspondence between these images and the real-time ultrasound 
! images. This step, discussed in detail later, need not be performed every time. Subsequently, in 
j step 205 the physician scans the area of interest of the human anatomy with the ultrasound probe 
115. For example, the physician may scan the brain of a patient to obtain ultrasound images of 
the brain. Next, in step 207 the computer overlays the pre-acquired images corresponding to 
other image modalities, such as CT and MR onto the ultrasound images. Later, in step 209 the 
I physician may move the surgical instruments or the ultrasound probe. Finally, in the sixth step 
: 21 1, the system can calculate and display the new position and orientation of the surgical 
\ instruments or the ultrasound probe on different images and also on three-dimensional images 
! overlaid on the real-time ultrasound images. 

i 

During surgery, the physician may go back to step 205 and scan another area of interest 
! and accordingly the computer may overlay pre-acquired images belonging to other image 
modalities onto the new ultrasound images. Alternatively, the physician may simply move the 

i 

ultrasound probe 1 15 or other surgical instruments as per step 209, and yet remain within the 
•i earlier scanned area of interest. Consequently, the system calculates the new location of the 
i ultrasound probe 1 15 or other surgical instruments and displays the overlapping images on the 

i 

display 121 attached to the system. 

Referring to Fig. 2, as discussed above, a physician or her assistant calibrates the 
navigational guidance system before using the system for the first time (step 201). In addition, 
the physician may need to calibrate the system later occasionally to improve the system's 
accuracy. Indeed, calibration may be performed by service technician familiar with the 

i 

i 

i 
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I navigational guidance system. Calibration is the process used to precisely calculate the location 

I of the ultrasound image plane relative to the localizer 116. 

j Calibration of image guided surgical devices is known in the art. In addition, a pending 

| U.S. Patent Application assigned to the same assignee, S/No. 09/274,972 filed on March 23, 

i 

1999 describes calibration in the context of a navigational guidance system using x-rays, which 
is incorporated by reference. In general, calibration consists of using a calibration device, which 
! may contain objects, such as rows and columns of wires inside a frame, that can be detected by 

! an ultrasound machine. A user scans the calibration device with the ultrasound probe 1 1 5 and 

I ... 

• the attached localizer 116. The ultrasound navigation system with the help of software running 

i 

on the computer 120 locates the intersection of wires, which appear as dots in the scanned image 

j 

: of the calibration device. The computer 120 then calculates a transformation between these dots 
j and the actual intersection of the wires that formed a dot located on the actual calibration device. 

As discussed earlier, a physician, or the physician's assistant, may use Computed 
j! Tomography (CT) or Magnetic Resonance Imaging (MRI) to acquire CT or MR data-sets 

corresponding to each one of these image modalities. These pre-acquired image data-sets can be 
| stored on a storage device coupled to the computer 120. In addition, these three-dimensional 
data sets may be acquired by the physician during the operation. Furthermore, the present 
invention is not limited to using CT or MR images. One may use images from other image 
modalities as well with the present invention. For example, images acquired using other 
diagnostic tools, such as Positron Emission Tomography (PET) or X-rays may also be used. 

With the ultrasound navigation system setup and the data-sets corresponding to other 
image modalities acquired, the physician may in step 205 begin surgery by scanning the specific 
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area of human anatomy which needs operative care with the ultrasound probe 115. The real-time 
image of the specific area acquired by the ultrasound probe 1 15 is displayed on the display which 
is part of the ultrasound machine. Next, upon the physician's command the system overlays the 
three-dimensional image corresponding to a different image modality data-set onto the 
ultrasound image along with an iconic representation of the surgical instrument 140. 

However, before overlaying the three-dimensional image with a graphical representation 
of a surgical instrument, the correspondence between points in the three-dimensional image and 
points in the patient's reference frame may need to be determined in step 203, as shown in Fig. 2. 
The process to achieve this correspondence is known as registration of the image. One method 
for performing image registration is described in the previously mentioned publications to 
Bucholz. Another method for performing image registration is described in a pending U.S. 
Patent Application to the same assignee, S/No. 09/274,972 filed on March 23, 1999. 

In general, three-dimensional patient specific images can be registered to a patient on the 
operating room table (surgical space) using multiple two-dimensional image projections. This 
process, generally referred to as 2D/3D registration, uses two pre-established spatial 
transformations to relate the surgical space to the three-dimensional image space. Accordingly, 
the first transformation is between the ultrasound images and the three-dimensional image data- 
set, such as a CT or an MR data-set corresponding to the same patient. 

The second transformation is between the coordinate system of the ultrasound images and 
an externally measurable reference system, which can be achieved using a tracking sensor i 1 7. 
Tracking sensor 1 17 is a real-time infrared tracking sensor linked to computer 120. Specially 
constructed surgical instruments and other markers, also known as localizers, in the field of 
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tracking sensor 1 17 can be detected and located in three-dimensional space. For example, a 
surgical instrument 140, such as a drill, may be embedded with tracking elements, such as for 
example, infrared emitters/reflectors on its handle. Tracking sensor 1 17 can then detect the 
presence and location of infrared emitters/reflectors. Because the relative spatial locations of the 
emitters/reflectors in instrument 140 are known a priori, tracking sensor 117 and computer 120 
are able to locate instrument 140 in three-dimensional space using known mathematical 
transformations. Instead of using infrared tracking sensor 117 and corresponding infrared 
emitters/reflectors, other types of positional location devices are known in the art, and may be 
used. For example, a positional location device may also be based on magnetic fields, optic 
emissions, sonic emissions, or radio waves. Once these two transformations have been 
established, the system may relate the surgical space directly to the three-dimensional image 
space. 

| Finally, with a display of the surgical instrument overlaid on the ultrasound image, which 

in turn is blended with a three-dimensional image from another modality the physician can guide 
the surgical instrument more effectively. Several well known techniques can be used to blend 

I the images. For example, a computer may implement an algorithm that for each pixel establishes 
a as the blending ratio. Assuming X is a pixel from the ultrasound image and Y is a pixel from 
the MR or CT image. Then the computer may perform the following calculation to arrive at a 
pixel Z for the blended image: 

Blended Image Pixel Z - ((X * a) + (Y*(l - a))) 

Referring to Figs. 1 and 5, when during surgery the physician moves the ultrasound probe 
1 15 or other surgical instrument(s) in step 509, the system calculates the new location of the 

15 



C c 

ultrasound probe 1 15 or the location of other surgical instruments. The system does this by using | 

i 

a technique called localization, which as discussed earlier, is known in the art. 

In addition to above methods and techniques a physician may achieve better surgical 
navigation using the techniques and methods discussed below. j 

i 

Displaying Surgical Instrument to Ultrasound Image Angle j 
Referring to Figs. 1, 3, and 4 the ultrasound navigation system may also be used to ! 
calculate and display the angle of a surgical instrument 340 to the ultrasound image plane 350. 
The computer 120 may calculate the angle by drawing a circle with its center at the point where 

j 

the surgical instrument's trajectory 342 passes through the ultrasound image plane 350. The 
radius of the circle may represent the angle of the trajectory 342 to the ultrasound plane 350. 
This is because the computer 120 may use trigonometric techniques such as drawing a 
perpendicular line from the tip 341 of the surgical instrument 340 onto the ultrasound image 
plane 350. Accordingly, the radius of a circle drawn by the computer 120 with its center at the 
point where the surgical instrument's trajectory 342 passes through the ultrasound image plane 
350 may represent the angle of the surgical instrument 340 to the ultrasound image plane 350. 
The system may display this angle on the display 121 as shown in Fig. 4. 

Referring to Fig. 4, the system may display a side view of the ultrasound image plane 
450. It may also display the trajectory of a surgical instrument 440 intersecting with the 
ultrasound image plane 450 and changing the display of the trajectory to a dotted representation 

i 

i 

442 after the trajectory crosses the ultrasound image plane 450. i 

i 

Fig. 5 is a pictorial image showing an MR data-set image 515 and an ultrasound image 

i 

530 side by side on the display 120. Both images have an iconic representation of a surgical 

16 
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instrument 505 superimposed on the images. The figure also indicates that the system displays a 
trajectory of a surgical instrument tip 507 and changes the trajectory from a solid line trajectory 
510 to a dashed line trajectory 515 when the trajectory crosses the MR data-set image plane and 
the ultrasound image plane respectively. Fig. 6 is another pictorial image representing a zoomed- 
in view of the MR image and the ultrasound image. A physician may select either view or may 
display both views on the display 121. 

Displaying Surgical Instrument to Ultrasound Image Angle Using Periodic Markers 
Referring to Fig. 7, the disclosed system can display the angle of the surgical instrument 
140 relative to the ultrasound image plane by displaying periodic markers 715. The distance 
between the periodic markers changes as a function of the cosine of the angle between the 
surgical instrument trajectory 710 and the ultrasound image plane. Accordingly, the periodic 
markers may also indicate the angle of the surgical instrument trajectory 710 to the ultrasound 
image. 

Displaying Orthogonal Images for Better Navigation 
Referring to Fig. 8, the ultrasound navigation system may also display multiple 
orthogonal ultrasound images 801 and 805. In addition, the system may also upon command 
display iconic representation 840 of a surgical instrument onto the orthogonal ultrasound images 
801 and 805. 

Overlaying Segmentations onto Ultrasound Images 
The system disclosed in Fig. 1, may also be used to overlay image segmentations onto 
ultrasound images for surgical navigation. In general, segmentations are surfaces or sets of other 
information extracted from a three-dimensional data-set. Segmentations can be obtained by the 
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computer 120, on command, from the pre-acquired three-dimensional image data-sets by using 

i 

known computer graphics algorithms. The extracted segmentation can then be overlaid onto an J 
ultrasound image in the correct perspective by the computer 120. The overlaid image can be j 

! 

displayed by the display 121, as discussed earlier. j 

I 

In addition, referring to Fig. 1, and as discussed earlier, the computer 120 also displays an J 

I 

iconic representation of any surgical instrument 140 with tracking sensors. Also, the computer 
120 can recalculate and display the location of the surgical instrument 140 onto the overlaid 
image when the physician moves the surgical instrument 140 during surgical navigation. 
Surgical Navigation with Three-dimensional Image Data-sets 
The ultrasound navigation system shown in Fig.l may also be used to help a physician in 
surgical navigation by displaying three-dimensional ultrasound image data-sets. A physician or a 
physician's assistant can acquire a three-dimensional ultrasound image data-set by using known 
techniques that may be free-hand or may involve use of mechanical movements. Once the image 
data is acquired using any of the known techniques, the computer 120 can then using graphics 
related mathematical computations reconstruct this data into an orthogonal data-set. Later, the 
computer 120 can display the orthogonal data-set as a three-dimensional image on the display 
121. 

The computer 120 then overlays a real-time ultrasound image over the pre-acquired and 
computationally generated three-dimensional image in the right perspective. In addition, the 
computer 120 with the help of known algorithms can map an ultrasound image relating to an area : 
of an interest onto the surface of the pre-acquired three-dimensional image of the same area. The 
general technique of mapping an image onto the surface of a three-dimensional image in the 
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correct perspective is known in the art, and is referrdil to as texture mapping. Accordingly, using 
known texture mapping algorithms, an ultrasound image of an area of interest can be mapped 
onto a three-dimensional image of a particular area of human anatomy, and can be displayed on 
the display 121, aiding a physician in better surgical navigation. 

In general; however, due to perspective transformation or the curvature of the surface, 
texture mapping may result in a distorted image. To alleviate this problem, the disclosed system 
may use known filtering-techniques that help reduce distortion. Thus, the computer 120 with the 
help of software can execute instructions corresponding to known filtering algorithms, which can 
reduce image distortion. 

In addition, referring to Fig. 1, and as discussed earlier, the computer 120 also displays an 
iconic representation of any surgical instrument 140 with tracking sensors. Also, the computer 
120 can recalculate and display the location of the surgical instrument 140 onto the texture 
mapped ultrasound image when the physician moves the surgical instrument 140 during surgical 
navigation. 

Detecting Organ Shift Using the Ultrasound Navigation System 
The ultrasound navigation system shown in Fig. 1 may also be used to detect organ- 
matter shift during surgery. Detection of organ-matter shift is important since a patient's organ- 
matter may have shifted from the time when CT or MR data-sets for that patient were created to 
the time when the patient is actually operated upon. For example, a physician's assistant may 
have obtained CT or MR images of a patient needing neurosurgery. However, the physician 
scheduled to operate upon the patient may not operate upon the patient until a later time during 
which the grey-matter or other organic matter comprising the brain may have shifted. Referring 
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to Fig. 9, once the ultrasound image 910 and the extracted CT or MR two-dimensional image 
950 are correlated the computer 120 with the help of software can perform certain calculations 
and measurements to indicate organ-matter shift. 

Correlation involves performing registration, localization, and calibration. Registration, 
as discussed earlier, involves determining the correspondence between points in the three- 
dimensional image and points in the patient's reference frame. One method for performing 
image registration is described in the previously mentioned publications to Bucholz. Another 
method for performing image registration is described in a pending U.S. Patent Application to 
the same assignee, S/No. 09/274,972 filed on March 23, 1999. 

In general, as discussed earlier, three-dimensional patient specific images can be 
registered to a patient on the operating room table (surgical space) using multiple two- 
dimensional image projections. This process, generally referred to as 2D/3D registration, uses 
two pre-established spatial transformations to relate the surgical space to the three-dimensional 
image space. Accordingly, the first transformation is between the ultrasound images and the 
three-dimensional image data-set, such as a CT or an MR data-set corresponding to the same 
patient. The second transformation is between the coordinate system of the ultrasound images 
and an externally measurable reference system which can be achieved using a tracking sensor 
117. The tracking sensor 1 17 is also known as a localizer array, and the process of achieving the 
previously mentioned transformation is referred to as localization. 

Tracking sensor 1 17 is a real-time infrared tracking sensor linked to computer 120. 

i 

Specially constructed surgical instruments and other markers in the field of tracking sensor 1 1 7 

i 

can be detected and located in three-dimensional space. For example, a surgical instrument 140, 
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i 

such as a drill, is embedded with infrared emitters/reflectors on its handle. Tracking sensor 1 17 
detects the presence and location of infrared emitters/reflectors. Because the relative spatial 
locations of the emitters/reflectors in instrument 1 40 are known a priori, tracking sensor 1 1 7 and 
computer 120 are able to locate instrument 140 in three-dimensional space using well known 
mathematical transformations. Instead of using infrared tracking sensor 1 17 and corresponding 
infrared emitters/reflectors, other types of positional location devices are known in the art, and 
may be used. For example, a positional location device may also be based on magnetic fields, 
j sonic emissions, or radio waves. Once these two transformations have been established, the 
system may relate the surgical space directly to the three-dimensional image space. 

Finally, the system may need to be calibrated. Calibration is the process to precisely 
calculate the location of the ultrasound image plane relative to the localizer 116. Consequently, 
' calibration improves the interpretation of the ultrasound images where the ultrasound probe 115 
; may have been displaced in relation to the localizer 116. 

j As discussed earlier, calibration of image guided surgical devices is known in the art. For 

example, the previously mentioned Bucholz references describe calibration. In addition, a 

! 

| pending U.S. Patent Application assigned to the same assignee, S/No. 09/274,972 filed on March 
23, 1999 describes calibration in the context of a navigational guidance system using x-rays. In 
general, calibration consists of using a calibration device, which may contain objects, such as 
rows and columns of wires inside a frame, that can be detected by an ultrasound machine. A user 
scans the calibration device with the ultrasound probe 115 and the attached localizer 116. The 
ultrasound navigation system with the help of software running on the computer 120 locates the 
intersection of wires, which appear as dots in the scanned image of the calibration device. The 
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computer 120 then calculates a transformation between these dots and the actual intersection of 
the wires that formed a dot located on the actual calibration device. 

Once the system disclosed in Fig. 1 has registered, calibrated, and localized the images, 
the system may be used to detect organ-matter shift using certain measurements. To perform 
these measurements, points 912, 914, and 916 must first be established within the ultrasound 
image 910. These points may be established using any peripheral device attached to the 

i 

computer 120. For example, a mouse attached to the computer 120 may be used to select the j 
points of interest on the ultrasound image 910. 

i 
I 

Once these points are selected the software on computer 120 can then transform the j 

i 

coordinates of these points into a three-dimensional image generated from CT or MR data-set 
specific to a patient. Next, the software running on computer 120 can extract the correlated CT \ 
or MR two-dimensional image from the three-dimensional CT or MR data-set. Having done so, 
the computer 120 can display the extracted two-dimensional image on the display 121 . 

The physician can then select points 952, 954, and 956 on the extracted two-dimensional 
image 950 corresponding to the ultrasound image 910 by finding the corresponding anatomy in 
the extracted two-dimensional image 950. The computer 120 can then display the distance and 
the trajectory 975 between the corresponding points, giving the physician a graphical view of the 
extent of the organ-matter shift. For example, referring to Fig. 6, the dotted line 975 represents 
the organ-matter shift as indicated by the distance between point 954 and point 954', which is 
where the corresponding point to 914 would have appeared had the organ-matter not shifted. j 

Overlay of Video Imagery onto Ultrasound Images j 

i 
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Referring to Fig. 10, an ultrasound machine 1050 is connected to a computer 1200 such 
that the acquired ultrasound images can be transferred to the computer 1200. The computer 1200 
is connected to a display 1210, such that images stored on a storage media can be digitally 
manipulated, saved, printed or displayed on the display 1210. Also, an X-ray machine or a 
laparoscope 1090 may also be connected to the computer such that X-ray images or video 
imagery from the laparoscope 1090 can be transferred to the computer 1200. In addition, like the 
ultrasound images, these images can also be digitally manipulated, saved, printed, or displayed 
on the display 1210. 

Referring to Fig. 10, both the ultrasound machine 1050 and the laparoscope 1090 have 
each a localizer 1 160, 1091 attached to them. As discussed earlier, the localizers could be, for 
example, either optical, electromagnetic, acoustic, or other known systems. In addition, 
i localization is a standard technique used in image guided surgery to locate the orientation and the 

! 
i 

| position of a surgical instrument, an ultrasound probe, or a laparoscope, relative to a patient's 

; position and is known in the art. As mentioned earlier, each of the Bucholz references discuss 

j 

! systems and methods that permit localization. 

Fig. 1 1 is a flow chart illustrating the steps for methods consistent with the present 
invention for ultrasound and video navigational guidance using the system of Fig. 10. As shown 
in Fig. 11, there are seven main steps involved in using the disclosed system. A physician needs 
to perform the first two steps: Calibrate ultrasound machine (step 1 100) and Calibrate 
laparoscope (step 1101), only occasionally. After calibration, which need not be performed 
every time, the physician in the third step, step 1 103, scans an area of interest of human anatomy 
using the ultrasound probe. Subsequently, in step 1 105 the physician obtains a video image of 
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the same area of that particular patient using a laparoscope 1090. Note that the physician may in 
this step use an X-ray machine to acquire an image of the same area of anatomy as well. In 
general, a physician may use any known system that permits the physician to obtain a video 
image of the relevant area of interest. 

Later, in step 1 107, the computer overlays the acquired ultrasound image onto the video 
image. For the images to be useful in surgical navigation the system ensures that the ultrasound 
probe 1 150 and the laparoscope 1090 are localized. As discussed earlier, localization is a known 

I 

technique. In the next step 1 109, the physician during surgery may move the ultrasound probe 

1 150, or the laparoscope 1090. In addition, the physician may be using a surgical instrument 

1400, which also has been localized, to perform the surgery. In step 1 1 1 1, the system calculates 

the new location of the surgical instrument 1400 and displays it on the overlaid images. If the 

physician, scans a different area of interest, then the system displays the overlaid images 
i m 
! corresponding to the new area of interest. In addition, the system calculates the new location of 

! 

I the surgical instrument 1400 and displays an iconic representation of the surgical instrument 

j 1400 on the display 1210. 

i 

j Figs. 12 and 13 are pictorial images representing the two video laparoscope images 1220 

i 

and 1320 of a gallbladder with an ultrasound image 1200/1300 overlaid in correct perspective. 
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